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What is an OSSE? 


An OSSE is a modeling experiment used to evaluate the impact 
of new observing systems on operational forecasts when 
actual observational data is not available. 


.A long free model run is used as the “truth” - the Nature Run 

.The Nature Run fields are used to back out “synthetic 
observations” from all current and new observing systems. 

.Suitable errors are added to the synthetic observations 

.The synthetic observations are assimilated into a different 
operational model 

.Forecasts are made with the second model and compared 
with the Nature Run to quantify improvements due to the new 
observing system 


OSSEs vs. the Real World 


Real world forecasts verification 
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Why do an OSSE? 


1 .You want to find out if a new observing system will 
add value to NWP analyses and forecasts 

2. You want to make design decisions for a new 
observing system 

3. You want to investigate the behavior of data 
assimilation systems in an environment where the 
truth is known 



When not to run an OSSE 


.When you can't model the phenomena you are 
interested in 

.When you can't simulate your new observations 

.When you can't assimilate your new 
observations 



Nature Runs 


. Nature Runs act as the 'truth' in the OSSE, 
replacing the real atmosphere. 

. Usually, a long free (non-cycling) forecast from the 
best available model is used as the NR 

. Model forecast has continuity of fields in time 

. Sometimes an analysis or reanalysis sequence is 
used, but the sequence of states of truth can never 
be replicated by a model 

. Always a push for bigger, higher resolution NR 



Nature Run Requirements 

.Must be able to realistically model phenomena 
of interest 

-Dynamics and physics should be realistic 
-Must produce fields needed for “observations” 

-Should be verified against real world 

.Ideally is ‘better’ than the operational model to 
be used for experiments 

.Preferably a different model base is used for the 
NR and the experimental forecast model to 
reduce incestuousness 
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G5 Nature Run 



2 year, 7 km/72L, 30 minute resolution 
15 aerosols, ozone, CO, C02 


Common Problems with Nature Runs 


.Nonexistence 

.Identical or fraternal twins 

.Outdated by the time you get to use them 

.Gigantic output files and huge computational 
resource requirements 

-Output saved at full spatial resolution but 30 min 
+ intervals 



Full resolution (1.5 km, 10 min) 
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Temporal interpolation (3 hrs) 
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Spatial interpolation to 27 km 
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Comparison of temporal and spatial 
Interpolation errors compared to 1.5 
km run for Typhoon Guchol (2012). 
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Nature Run Validation 


.Evaluate if NR is sufficiently realistic to yield 
meaningful results 

.In addition to the phenomena of interest, the NR 
needs to realistically replicate fields needed to 
generate synthetic observations 

.Can't validate everything; corollary - don't 
expect a NR to come pre-validated for your 
needs 
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Latitude of maximum monthly mean zonal mean streamfunction 
CFSR 1994-2010, blue; ERA-INT 1994-2013, green; NR, red 
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Synthetic Observations 


Example of AIRS observations channel 295 at 18 UTC 12 July 
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Observation Errors 


.Synthetic observations contain some intrinsic 
interpolation/operator errors, but less than real 
observations (usually) 

.Synthetic errors are created and added to the 
synthetic observations to compensate 

.Error is complex and poorly understood 

.Error magnitude 
.Biases 

.Correlated errors 



Calibration 


.Adjust synthetic observations and their errors to 
increase realism of the OSSE in a statistical 
sense 

.Compare OSSE statistics to statistics using real 
data in the same DAS/forecast system 

.Need to decide what statistical metrics to use for 
the calibration, depending on your needs 

.Calibrating new observation types? 

.Find an analogous data type if possible 



Count 


AMSU-a n19 count 



Channel 


Observation count is easy to calibrate 



AMSU-A 


RAOB Wind 



O-F is fairly easy to calibrate because you can 
manipulate O directly. 
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AIRS Channel Correlations, Real 


OSSE AIRS channel correlations 
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Some observation correlations are relatively 
easy to calibrate 




Real OSSE 



Latitude Latitude 


A-B (analysis increment) is a little harder to 
calibrate, as A and B are not directly controlled 


U, 30N-60N 


U, 30S-60S 


U, Tropics 



RMSE 



RMSE RMSE 


Forecast errors are harder to calibrate, 
especially for longer forecasts. Matching of 
this statistic by manipulation of observations is 
difficult to impossible beyond ~24 hour 
forecasts. 
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Northern Hemisphere 
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Southern Hemisphere 
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Why believe OSSE results? 
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New observations can be put into context relative to 
existing observation impacts 


Adjoint Observation Impact 



Criticisms of OSSEs 


.Results only apply within the OSSE system - no 
concrete connection to the real world 

.Even the best OSSEs are far from perfect: 
incestuousness, difficulty in generating 
observations and errors, deficiencies of the 
Nature Run 

.By the time the new instrument is deployed, both 
the global observing network and the forecast 
models/DAS will be different 

.Examples of sloppy or unsuccessful OSSEs 



Common Pitfalls 


.Very reduced baseline of assimilated 
observational data (ex. no radiance data) 

.Other artificial degradation of analysis state 

.No validation or calibration of OSSE framework 

.Obtaining robust results from case studies is 
very challenging 

-Use ensemble forecasts if you can! 



Choosing Metrics 


.Long cycling periods necessary to get 
statistically significant results for most new 
observations 

.Anomaly correlation is a difficult metric to show 
appreciable impacts 

.What fields do you expect the instrument to 
improve? 

.Largest impacts found at analysis time or short- 
term forecasts 



Idealized Studies 


.Identical twin experiments 
.Idealized observations 
.Manipulation of observation errors 
.Experiments with B, R 

.Make use of available “Truth” 



Takeaway 


.OSSEs can provide useful information about 
new observational types and the workings of data 
assimilation systems 

.Careful consideration of research goals should 
guide each step of the OSSE process 

.OSSEs are hard, good OSSEs are harder 



